2023-10-19
While historically biologically-inspired, they are very far from a biological neuron
Point neurons are abstraction built to answer specific questions at a specific time
From a ‘fit to neuroscience data’ point-of-vue, it’s a bad model BUT
The point neuron has found its usefullness since it has become the default unit model in ANN
Most of the complexity is abstracted away for better computational efficiency.
But this neuron abstraction coupled with the way we train deep nets has shown cases where they fail miserably
→ continual learning and the catastrophic forgetting problem
Multiple studies by many teams has point toward the importance of complexity both at the level of single neuron and in neural networks.
Examples (non-exhaustive):
Payeur et al. (2021) : Burst-dependent synaptic plasticity can coordinate learning in hierarchical circuits
when paired with short-term synaptic dynamics, regenerative activity in the apical dendrites, and synaptic plasticity in feedback pathways, a burst-dependent learning rule can solve challenging tasks that require deep network architectures. […]well-known properties of dendrites, synapses, and synaptic plasticity are sufficient to enable sophisticated learning in hierarchical circuits.
Perez-Nieves et al. (2021): Neural heterogeneity promotes robust learning
the performance of spiking neural networks trained to carry out tasks of real-world difficulty, with varying degrees of heterogeneity, and found that it substantially improved task performance.
Zeldenrust, Gutkin, and Denéve (2021): Efficient and robust coding in heterogeneous recurrent networks
networks consisting of a heterogeneous population of different neuron types are both more efficient and more robust against correlated noise.
Jones and Kording (2021): Might a Single Neuron Solve Interesting Machine Learning Problems Through Successive Computations on Its Dendritic Tree?
simple model where the dendrite is implemented as a sequence of thresholded linear units. We manipulate the architecture of this model to investigate the impacts of binary branching constraints and repetition of synaptic inputs on neural computation. […] models with such manipulations can perform well on machine learning tasks, such as Fashion MNIST or Extended MNIST. We find that model performance on these tasks is limited by binary tree branching and dendritic asymmetry and is improved by the repetition of synaptic inputs to different dendritic branches.
Dendrites are major elements of biological neurons
They are projections from the cell’s soma into the surrounding environment
They make up most of the substrate of connections between cells (there can also be sub-structures called spines or not)
Dendrites have extremely diverse morphologies
(Beniaguev, Segev, and London 2021)
However, no matter how deep an ANN is, it will always be subject to catastrophic forgetting.
Hypothesis : Dendritic units can help minimize the catastrophic forgetting problem in continual learning while still maintaining a low compute overhead
Dendrite can act as coincidence detectors.
Differing information content is routed to different part of the dendritic arbor
Leading to the potential emergence of functional subdomains
Neurons contains active or passive dendrites.
Passive dendrites show only linear or sublinear input summation.
Active dendrite can show supralinear outpout.
Same dendritic branch stimulation can results in linear or supra-linear output depending on stimulation conditions.
Case study 1: Tzilivaki, Kastellakis, and Poirazi (2019) - Challenging the point neuron dogma: FS basket cells as 2-stage nonlinear integrators
Case study 2: Jones and Kording (2021) - Might a Single Neuron Solve Interesting Machine Learning Problems Through Successive Computations on Its Dendritic Tree?
Modelling dendrites as a sequence of thresholded linear unit solves typical ML benchmarks (non-CL scenarios)
Benefit from having the same input onto several branches of the dendritic tree = importance of synaptic input repetition
Performances on-par with a fully-connected neural network (FCNN)
Case study 3: Grewal et al. (2021) - Going Beyond the Point Neuron: Active Dendrites and Sparse Representations for Continual Learning and Iyer et al. (2022) - Avoiding Catastrophe: Active Dendrites Enable Multi-Task Learning in Dynamic Environments
Adding dendrite abstractions in an MLP help preventing catastrophic forgetting both in abstract task (permutedMNIST) and reinforcement learning (RL) tasks.
Context-driven task-specific subnetworks activation gated by dendritic inputs
Case study 3: Grewal et al. (2021) - Going Beyond the Point Neuron: Active Dendrites and Sparse Representations for Continual Learning and Iyer et al. 2022 - Avoiding Catastrophe: Active Dendrites Enable Multi-Task Learning in Dynamic Environments
Dendrite specialize in few tasks.
Contradict the hypothesis that a dendritic neuron is equivalent to a multilayer network as classical deep networks are incapable of performing well due to catastrophic forgetting, regardless of network depth
Dendrites’ computational role is far from solved
Neuroscience informs models design which in turn inform back neuroscience. That’s a neat loop
Depending upon the task, adding dendrites could increase computational overhead without benefits i.e. in non continual learning scenario there might be few, if any, benefits to adding dendrites
In continual learning settings, dendrites could be a powerfull mechanism to design networks more robust to catastrophic forgetting
There is still a lot to open ended questions here.